A live notebook of working examples of using Sinar's Popit API and database of Malaysian MPs.
Detailed information of Persons should probably be appended to post memberships this would allow us to show post (Seat) information, not just details of person.
Refactor functions here into common library
Feel free to do pull requests, or contact me on issues with the data.
Khairil Yusof khairil.yusof@sinarproject.org
In [1]:
import requests
import json
#Dewan Rakyat MP Posts in Sinar Malaysia Popit Database
posts = []
for page in range(1,10):
dewan_rakyat_request = requests.get('http://sinar-malaysia.popit.mysociety.org/api/v0.1/search/posts?q=organization_id:53633b5a19ee29270d8a9ecf'+'&page='+str(page))
for post in (json.loads(dewan_rakyat_request.content)['result']):
posts.append(post)
Now we will load up information on the MPs holding these posts
In [2]:
import datetime
from dateutil import parser
current = datetime.date(2013,5,5)
def has_end_date(member):
if member.has_key('end_date') and member['end_date'] == '':
return False
elif not member.has_key('end_date'):
return False
else:
return True
def current_MP(member):
#Legislative tag term here would simply
if (parser.parse(member['start_date'])).date() > current:
if not has_end_date(member):
return True
else:
return False
def person(person_id):
#Load up information of persons from Popit database
req = requests.get('https://sinar-malaysia.popit.mysociety.org/api/v0.1/persons/' + person_id)
return json.loads(req.content)['result']
def age(str):
#calculate age based on date strings stored in Popit
born = parser.parse(str)
today = datetime.date.today()
age = today.year - born.year - ((today.month, today.day) < (born.month, born.day))
return int(age)
#Current MPs will not have end dates, and have terms after 2013-05-05
MP_ids = []
for post in posts:
for member in post['memberships']:
if current_MP(member):
MP_ids.append(member['person_id'])
In [4]:
#Pull down the data of current MPs from Popit Database add calculate age if there is birthdate
MPs = []
for id in MP_ids:
MPs.append(person(id))
In [23]:
for MP in MPs:
if MP.has_key('birth_date'):
if MP['birth_date']:
#add current age in addition to the values in Popit
MP['age'] = age(MP['birth_date'])
In [6]:
WomenMPs = []
for MP in MPs:
if MP.has_key('gender') and MP['gender'] == 'Female':
WomenMPs.append(MP)
print "Number of Women MPs " + str(len(WomenMPs))
for MP in WomenMPs:
print MP['name']
Number of Women MPs 24
Norah Abdul Rahman
Nurul Izzah Anwar
Teresa Kok Suh Sim
Mas Ermieyati Samsudin
Noraini Ahmad
Halimah Mohamed Sadique
Azalina Othman Said
Normala Abdul Samad
Teo Nie Ching
Rosnah Abdul Rashid Shirlin
Azizah Mohd Dun
Tsen Thau Lin
Yap Kain Ching
Julian Tan Kok Ping
Rubiah Wang
Nancy Shukri
Rohani Abdul Karim
Alice Lau Kiong Yieng
Wan Azizah Wan Ismail
Noriah Kasnon
Siti Zailah Binti Mohd Yusoff
Kasthuriraani Patto
Fuziah Binti Salleh
Zuraida Kamaruddin
In [25]:
import numpy
#list of ages
ages = []
for MP in MPs:
if MP.has_key('age'):
ages.append(int(MP['age']))
print numpy.median(ages)
print numpy.max(ages)
print numpy.min(ages)
56.0
74
32
pandas is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language.
If you're learning Python to work with data, it's worth getting used to this library, as it provides pretty much all you will need when working with data from importing and cleaning messy data, to exporting it, including working with very large data sets.
A lot of the earlier work, such as cleaning, getting unique values etc. could be done easily with built-in functions of pandas as a DataFrame.
In [26]:
import pandas
pandas.DataFrame(MPs)
Out[26]:
age
birth_date
contact_details
death_date
dissolution_date
end_date
founding_date
gender
html_url
id
...
initials
links
memberships
name
other_names
proxy_image
sources
start_date
summary
url
0
39
1975-09-12
[{u'type': u'phone', u'id': u'558337eef3c3651a...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e48755222837c2c0596a7
...
NaN
[{u'url': u'http://ongkianming.com', u'note': ...
[{u'contact_details': [], u'links': [], u'end_...
Ong Kian Ming
[{u'name': u'王建民', u'id': u'558337eef3c3651a49...
https://sinar-malaysia.popit.mysociety.org/ima...
[{u'url': u'http://ongkianming.com', u'note': ...
NaN
Dr. Ong Kian Ming was elected as a first-time ...
https://sinar-malaysia.popit.mysociety.org/api...
1
54
1960-11-01
[{u'type': u'phone number', u'id': u'5590d64dd...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e46ee5222837c2c0592eb
...
NaN
[{u'url': u'http://charlessantiago.org/', u'no...
[{u'contact_details': [], u'links': [], u'end_...
Charles Anthony A/L R. Santiago
[{u'name': u'Charles Santiago', u'id': u'55918...
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/api...
2
46
1968-12-18
[{u'type': u'email', u'id': u'55b351e0f1faa736...
NaN
NaN
NaN
NaN
Male
https://sinar-malaysia.popit.mysociety.org/per...
53635149f1eab6270da6c8f6
...
NaN
[{u'url': u'https://en.wikipedia.org/wiki/Wong...
[{u'area_id': None, u'contact_details': [], u'...
Wong Chen
[{u'name': u'Wong Chen X', u'id': u'55b351e0f1...
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
Wong Chen is a Malaysian politician and is the...
https://sinar-malaysia.popit.mysociety.org/api...
3
55
1959-10-04
[{u'type': u'address', u'id': u'55b6017353d566...
NaN
NaN
NaN
NaN
Female
https://sinar-malaysia.popit.mysociety.org/per...
545e43f15222837c2c058b76
...
NaN
[{u'url': u'http://dtknorah.blogspot.com/', u'...
[{u'contact_details': [], u'links': [], u'end_...
Norah Abdul Rahman
[{u'note': u'With honorifics', u'name': u'Datu...
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
Datuk Hajjah Norah Abdul Rahman (born 4 Octobe...
https://sinar-malaysia.popit.mysociety.org/api...
4
37
1977-09-14
[{u'type': u'handphone number', u'id': u'5582d...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e429d5222837c2c058829
...
NaN
[{u'url': u'https://ms-my.facebook.com/rafizir...
[{u'contact_details': [], u'links': [], u'end_...
Mohd Rafizi Bin Ramli
[]
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/api...
5
64
1951-07-16
[{u'type': u'facebook', u'id': u'5582d9c500912...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e45785222837c2c058f42
...
NaN
[{u'url': u'https://en.wikipedia.org/wiki/Che_...
[{u'contact_details': [], u'links': [], u'end_...
Che Rosli Bin Che Mat
[{u'name': u'Dr. Che Rosli Che Mat', u'id': u'...
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/api...
6
42
1973-06-19
[{u'type': u'phone', u'id': u'5582db444a70d893...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e43a65222837c2c058abc
...
NaN
[{u'url': u'https://en.wikipedia.org/wiki/Gobi...
[{u'contact_details': [], u'links': [], u'end_...
Gobind Singh Deo
[]
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
Gobind Singh Deo (born 19 June 1973) is a prom...
https://sinar-malaysia.popit.mysociety.org/api...
7
45
1970-06-18
[{u'type': u'facebook', u'id': u'55b5fcce9fcfa...
NaN
NaN
NaN
NaN
Male
https://sinar-malaysia.popit.mysociety.org/per...
545e44d15222837c2c058da1
...
NaN
[]
[{u'contact_details': [], u'links': [], u'end_...
Hee Loy Sian
[]
https://sinar-malaysia.popit.mysociety.org/ima...
[{u'url': u'http://www.keadilandaily.com', u'n...
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/api...
8
43
1972-08-01
[{u'type': u'twitter', u'id': u'55830e49f3c365...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e469b5222837c2c05921b
...
NaN
[{u'url': u'http://tonypua.blogspot.com', u'no...
[{u'contact_details': [], u'links': [], u'end_...
Tony Pua Kiam Wee
[{u'name': u'Pua Kiam Wee', u'id': u'55830e49f...
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/api...
9
58
1956-12-08
[{u'type': u'phone', u'id': u'55b6edf00d1ed7f9...
NaN
NaN
NaN
NaN
Male
https://sinar-malaysia.popit.mysociety.org/per...
545e45805222837c2c058f56
...
NaN
[{u'url': u'http://sivarasa.blogspot.com', u'n...
[{u'contact_details': [], u'links': [], u'end_...
Sivarasa Rasiah
[{u'name': u'Sivarasa A/L K. Rasiah', u'id': u...
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
Sivarasa Rasiah (Tamil: சிவராசா ராசையா), or al...
https://sinar-malaysia.popit.mysociety.org/api...
10
57
1957-08-14
[{u'type': u'twitter', u'id': u'55832f5e80ad1f...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e43335222837c2c05899c
...
NaN
[{u'url': u'http://www.khalidsamad.com', u'not...
[{u'contact_details': [], u'links': [], u'end_...
Khalid Abdul Samad
[{u'name': u'Hj Khalid Abd Samad', u'id': u'55...
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
A politician who is a member of PAS. He gradua...
https://sinar-malaysia.popit.mysociety.org/api...
11
NaN
NaN
[{u'type': u'handphone number', u'id': u'55833...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e43145222837c2c058952
...
NaN
[]
[{u'contact_details': [], u'links': [], u'end_...
G Manivannan A/L Gowindasamy
[]
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/api...
12
57
1958-02-14
[{u'type': u'phone number', u'id': u'55833b42f...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e44185222837c2c058bd8
...
NaN
[{u'url': u'drsitimariah.blogspot.com/', u'not...
[{u'contact_details': [], u'links': [], u'end_...
Siti Mariah Binti Mahmud
[{u'name': u'Dr. Siti Mariah binti Mahmud', u'...
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/api...
13
NaN
NaN
[{u'type': u'address', u'id': u'5571b6775ccbb5...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e447a5222837c2c058cca
...
NaN
[]
[{u'contact_details': [], u'links': [], u'end_...
Abdullah Sani Abdul Hamid
[{u'name': u'Abdullah Sani Bin Abdul Hamid', u...
https://sinar-malaysia.popit.mysociety.org/ima...
[{u'url': u'http://www.mtuc.org.my/about-us/of...
NaN
Abdullah Sani Abdul Hamid is a Malaysian polit...
https://sinar-malaysia.popit.mysociety.org/api...
14
NaN
NaN
[{u'type': u'twitter', u'id': u'5571b661e5ce8b...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e45f95222837c2c059086
...
NaN
[{u'url': u'https://ybhanipamaidin.wordpress.c...
[{u'contact_details': [], u'links': [], u'end_...
Mohamed Hanipa Maidin
[{u'name': u'Hanipa Maidin', u'id': u'5571b661...
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/api...
15
73
1942-05-26
[{u'type': u'twitter', u'id': u'5571b64bba593e...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e47f15222837c2c05956b
...
NaN
[{u'url': u'http://tansenggiaw.blogspot.com/',...
[{u'contact_details': [], u'links': [], u'end_...
Tan Seng Giaw
[{u'name': u'Tan Seng Giaw @ Tan Chun Tin', u'...
https://sinar-malaysia.popit.mysociety.org/ima...
[{u'url': u'http://www.thenutgraph.com/mp-watc...
NaN
Dr. Tan Seng Giaw (Chinese : 陈胜尧;) is a Malays...
https://sinar-malaysia.popit.mysociety.org/api...
16
51
1963-12-21
[{u'type': u'twitter', u'id': u'5571b61f46ee15...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
556c804aaf36fc0760ed58cf
...
[]
[{u'contact_details': [], u'links': [], u'end_...
Chua Tian Chang
[{u'name': u'Tian Chua', u'id': u'5571b61f46ee...
https://sinar-malaysia.popit.mysociety.org/ima...
[]
NaN
Chua Tian Chang, better known as Tian Chua (Si...
https://sinar-malaysia.popit.mysociety.org/api...
17
43
1972-07-25
[{u'type': u'facebook', u'id': u'5571b5ef46ee1...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e472b5222837c2c059384
...
NaN
[{u'url': u'http://limlipeng.blogspot.com/', u...
[{u'contact_details': [], u'links': [], u'end_...
Lim Lip Eng
[]
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
Lim Lip Eng is a Malaysian politician and is t...
https://sinar-malaysia.popit.mysociety.org/api...
18
52
1963-06-02
[{u'type': u'address', u'id': u'5571b5cfe5ce8b...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e45d25222837c2c059023
...
NaN
[]
[{u'contact_details': [], u'links': [], u'end_...
Ahmad Fauzi Zahari
[{u'note': u'With honorifics', u'name': u'Dato...
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/api...
19
NaN
NaN
[{u'type': u'facebook', u'id': u'5571b5b6e5ce8...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e470b5222837c2c059333
...
NaN
[]
[{u'contact_details': [], u'links': [], u'end_...
Johari Abdul Ghani
[{u'note': u'With honorifics', u'name': u'Datu...
https://sinar-malaysia.popit.mysociety.org/ima...
[{u'url': u'http://uda.com.my/about/#lead', u'...
NaN
Dato' Haji Johari Abdul Ghani is Barisan Nasio...
https://sinar-malaysia.popit.mysociety.org/api...
20
NaN
NaN
[{u'type': u'address', u'id': u'5571b58d46ee15...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e42945222837c2c058813
...
NaN
[{u'url': u'http://fongkuilun.blogspot.com/', ...
[{u'contact_details': [], u'links': [], u'end_...
Fong Kui Lun
[]
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
Fong Kui Lun (Chinese: 方贵伦) is a Malaysian pol...
https://sinar-malaysia.popit.mysociety.org/api...
21
34
1980-11-19
[{u'type': u'address', u'id': u'55b5fd3c9fcfab...
NaN
NaN
NaN
NaN
Female
https://sinar-malaysia.popit.mysociety.org/per...
545e469e5222837c2c059223
...
NaN
[{u'url': u'http://www.nurulizzah.com', u'note...
[{u'contact_details': [], u'links': [], u'end_...
Nurul Izzah Anwar
[{u'name': u'Nurul Izzah Binti Anwar', u'id': ...
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
A Malaysian politician from Parti Keadilan Rak...
https://sinar-malaysia.popit.mysociety.org/api...
22
51
1964-03-31
[{u'type': u'facebook', u'id': u'55b5fd569fcfa...
NaN
NaN
NaN
NaN
Female
https://sinar-malaysia.popit.mysociety.org/per...
545e46185222837c2c0590d5
...
NaN
[{u'url': u'http://teresakok.com/', u'note': u...
[{u'contact_details': [], u'links': [], u'end_...
Teresa Kok Suh Sim
[]
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
Teresa Kok Suh Sim (Chinese: 郭素沁) is a Malaysi...
https://sinar-malaysia.popit.mysociety.org/api...
23
57
1957-10-07
[{u'type': u'facebook', u'id': u'5571b54a46ee1...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e425c5222837c2c058787
...
NaN
[{u'url': u'Official Website', u'note': u'http...
[{u'contact_details': [], u'links': [], u'end_...
Tan Kok Wai
[]
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
Tan Kok Wai (born 7 October 1957) is a Malaysi...
https://sinar-malaysia.popit.mysociety.org/api...
24
68
1946-12-14
[{u'type': u'twitter', u'id': u'5571b52b46ee15...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e444f5222837c2c058c61
...
NaN
[]
[{u'contact_details': [], u'links': [], u'end_...
Abdul Khalid Ibrahim
[{u'name': u'Khalid Ibrahim', u'id': u'5571b52...
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
Tan Sri Dato' Seri Abdul Khalid bin Ibrahim is...
https://sinar-malaysia.popit.mysociety.org/api...
25
NaN
NaN
[{u'type': u'facebook', u'id': u'5571b4fe46ee1...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e44215222837c2c058bef
...
NaN
[]
[{u'contact_details': [], u'links': [], u'end_...
Zainudin Ismail
[{u'note': u'With honorifics', u'name': u'Dato...
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/api...
26
NaN
NaN
[{u'type': u'facebook', u'id': u'5571b4e246ee1...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e46c35222837c2c05927f
...
NaN
[]
[{u'contact_details': [], u'links': [], u'end_...
Mohamed Isa Abdul Samad
[{u'note': u'With honorifics', u'name': u'Tan ...
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
YB Tan Sri Haji Mohd Isa Dato' Haji Abdul Sama...
https://sinar-malaysia.popit.mysociety.org/api...
27
38
1977-04-28
[{u'type': u'facebook', u'id': u'5571b4a946ee1...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e448c5222837c2c058cf5
...
NaN
[{u'url': u'http://lokesiewfook.blogspot.com/'...
[{u'contact_details': [], u'links': [], u'end_...
Loke Siew Fook
[{u'name': u'Anthony Loke Siew Fook', u'id': u...
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
Anthony Loke Siew Fook is a Malaysian politici...
https://sinar-malaysia.popit.mysociety.org/api...
28
69
1946-04-24
[{u'type': u'facebook', u'id': u'5571b48c46ee1...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e45905222837c2c058f7f
...
NaN
[]
[{u'contact_details': [], u'links': [], u'end_...
Hasan Malek
[{u'note': u'With honorifics', u'name': u'Dato...
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
Dato' Sri Haji Hasan Malek is a Malaysian poli...
https://sinar-malaysia.popit.mysociety.org/api...
29
NaN
NaN
[{u'type': u'facebook', u'id': u'5571b47646ee1...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e47505222837c2c0593e3
...
NaN
[{u'url': u'http://teokokseong.blogspot.com/',...
[{u'contact_details': [], u'links': [], u'end_...
Teo Kok Seong
[]
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
Teo Kok Seong is a Malaysian politician who is...
https://sinar-malaysia.popit.mysociety.org/api...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
192
50
1964-10-25
[{u'note': u'MyMP Survey', u'type': u'phone nu...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e46ac5222837c2c059246
...
NaN
[{u'url': u'http://drmujahid.com', u'note': u'...
[{u'contact_details': [], u'links': [], u'end_...
Dr. Mujahid Yusof Rawa
[{u'name': u'Mujahid Yusof Rawa', u'id': u'55a...
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/api...
193
NaN
NaN
[{u'type': u'phone number', u'id': u'557bf846d...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e42c25222837c2c058883
...
NaN
[{u'url': u'https://www.facebook.com/dapthomas...
[{u'contact_details': [], u'links': [], u'end_...
Su Keong Siong
[]
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/api...
194
NaN
NaN
[{u'type': u'phone number', u'id': u'5575c5775...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e42d85222837c2c0588bc
...
NaN
[{u'url': u'https://www.facebook.com/reezal.me...
[{u'contact_details': [], u'links': [], u'end_...
Reezal Merican Bin Naina Merican
[]
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/api...
195
58
1957-03-12
[{u'type': u'phone number', u'id': u'55a36cb47...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e475c5222837c2c0593ff
...
NaN
[]
[{u'contact_details': [], u'links': [], u'end_...
Hamzah Zainuddin
[]
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/api...
196
48
1967-01-06
[{u'type': u'phone number', u'id': u'557b34987...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e425e5222837c2c05878d
...
NaN
[]
[{u'contact_details': [], u'links': [], u'end_...
Shamsul Anuar Bin Nasarah
[{u'name': u'Datuk Samsul Anuar Nasarah', u'id...
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/api...
197
NaN
NaN
[{u'type': u'phone number', u'id': u'557b0ada7...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e47375222837c2c0593a3
...
NaN
[]
[{u'contact_details': [], u'links': [], u'end_...
Hasbullah Bin Osman
[{u'name': u'Datuk Hasbullah Osman', u'id': u'...
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/api...
198
NaN
NaN
[{u'type': u'handphone', u'id': u'557a9c0d435e...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e45725222837c2c058f33
...
NaN
[]
[{u'contact_details': [], u'links': [], u'end_...
N Surendran A/L K Nagarajan
[]
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/api...
199
63
1952-07-21
[{u'type': u'phone number', u'id': u'55bdb7fc1...
NaN
NaN
NaN
NaN
Male
https://sinar-malaysia.popit.mysociety.org/per...
545e42e25222837c2c0588d4
...
NaN
[]
[{u'contact_details': [], u'links': [], u'end_...
Ahmad Husni Mohamad Hanadzlah
[{u'name': u'Datuk Ahmad Husni Hanadzlah', u'i...
https://sinar-malaysia.popit.mysociety.org/ima...
[{u'url': u'http://www.bloomberg.com/research/...
NaN
Dato’ Seri Ahmad Husni Mohamad Hanadzlah was a...
https://sinar-malaysia.popit.mysociety.org/api...
200
60
1955-03-28
[{u'type': u'phone number', u'id': u'55aa504af...
NaN
NaN
NaN
NaN
Male
https://sinar-malaysia.popit.mysociety.org/per...
545e44e35222837c2c058dcd
...
NaN
[{u'url': u'http://en.wikipedia.org/wiki/Micha...
[{u'contact_details': [], u'links': [], u'end_...
Michael Jeyakumar Devaraj
[{u'name': u'Dr Michael Jeyakumar Devaraj', u'...
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
Michael Jeyakumar Devaraj is a Malaysian polit...
https://sinar-malaysia.popit.mysociety.org/api...
201
NaN
None
[{u'type': u'phone number', u'id': u'55a37b76b...
NaN
NaN
NaN
NaN
Male
https://sinar-malaysia.popit.mysociety.org/per...
557b39bdeb081bf345655519
...
[{u'url': u'https://idrisahmad.wordpress.com/a...
[{u'contact_details': [], u'links': [], u'url'...
Idris Ahmad
[{u'name': u'Idris Ahmad', u'id': u'55a37b76ba...
https://sinar-malaysia.popit.mysociety.org/ima...
[]
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/api...
202
61
1954-05-15
[{u'type': u'phone number', u'id': u'55a37dd5f...
NaN
NaN
NaN
NaN
Male
https://sinar-malaysia.popit.mysociety.org/per...
545e43835222837c2c058a67
...
NaN
[]
[{u'contact_details': [], u'links': [], u'end_...
Mohd Nazri Abdul Aziz
[{u'name': u'Mohamed Nazri Bin Abdul Aziz', u'...
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/api...
203
53
1961-09-29
[{u'type': u'phone number', u'id': u'557c57e5d...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e44005222837c2c058b9d
...
NaN
[{u'url': u' ngehkooham.blogspot.com', u'note'...
[{u'contact_details': [], u'links': [], u'url'...
Ngeh Koo Ham
[]
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/api...
204
NaN
NaN
[]
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e489c5222837c2c059705
...
NaN
[]
[{u'contact_details': [], u'links': [], u'post...
Sivakumar A/L Varatharaju Naidu
[]
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/api...
205
42
1972-11-11
[{u'type': u'phone number', u'id': u'558f1e0eb...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e45745222837c2c058f39
...
NaN
[{u'url': u'https://twitter.com/ngakorming?lan...
[{u'contact_details': [], u'links': [], u'end_...
Nga Kor Ming
[]
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/api...
206
53
1961-11-28
[{u'type': u'phone number', u'id': u'55a9c7453...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e423f5222837c2c058741
...
NaN
[]
[{u'contact_details': [], u'links': [], u'end_...
Noor Azmi Bin Ghazali
[]
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/api...
207
55
1960-01-16
[{u'type': u'phone number', u'id': u'557c554b7...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e43b45222837c2c058adf
...
NaN
[]
[{u'contact_details': [], u'links': [], u'end_...
Wan Mohammad Khair-Il Anuar Bin Wan Ahmad
[]
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
An architect by profession. He owns an archite...
https://sinar-malaysia.popit.mysociety.org/api...
208
61
1954-07-10
[{u'type': u'phone number', u'id': u'55af09332...
NaN
NaN
NaN
NaN
Male
https://sinar-malaysia.popit.mysociety.org/per...
545e48c25222837c2c059765
...
NaN
[]
[{u'contact_details': [], u'links': [], u'url'...
Mohamad Imran Bin Abd Hamid
[]
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
A retired naval officer, now a politician and ...
https://sinar-malaysia.popit.mysociety.org/api...
209
47
1968
[{u'type': u'phone number', u'id': u'55ac6af20...
NaN
NaN
NaN
NaN
Male
https://sinar-malaysia.popit.mysociety.org/per...
545e42955222837c2c058816
...
NaN
[{u'url': u'https://www.facebook.com/KoChungSe...
[{u'contact_details': [], u'links': [], u'end_...
Ko Chung Sen
[]
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/api...
210
67
1948-01-17
[{u'type': u'phone number', u'id': u'557c6fe3d...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e47b85222837c2c0594e2
...
NaN
[]
[{u'contact_details': [], u'links': [], u'end_...
Tajuddin Bin Abdul Rahman
[{u'name': u'Datuk Tajuddin Abdul Rahman', u'i...
https://sinar-malaysia.popit.mysociety.org/ima...
[{u'url': u'http://www.freemalaysiatoday.com/w...
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/api...
211
NaN
NaN
[]
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e452b5222837c2c058e80
...
NaN
[]
[{u'contact_details': [], u'links': [], u'post...
Madhi Bin Hasan
[]
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/api...
212
NaN
NaN
[{u'type': u'phone number', u'id': u'557c5bc0d...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e461b5222837c2c0590dc
...
NaN
[]
[{u'contact_details': [], u'links': [], u'end_...
Mohd Zaim Bin Abu Hasan
[]
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/api...
213
55
1959-09-26
[{u'type': u'phone', u'id': u'55ad8b0a5ebd2964...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e48fe5222837c2c0597f8
...
NaN
[{u'url': u'https://www.facebook.com/leeboonch...
[{u'contact_details': [], u'links': [], u'end_...
Lee Boon Chye
[{u'name': u'Dr Lee Boon Chye', u'id': u'55ad8...
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/api...
214
NaN
NaN
[{u'type': u'phone number', u'id': u'557c690e2...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e48a75222837c2c059720
...
NaN
[]
[{u'contact_details': [], u'links': [], u'url'...
Saravanan A/L Murugan
[]
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/api...
215
NaN
NaN
[{u'type': u'phone number', u'id': u'557db9903...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e44d65222837c2c058dac
...
NaN
[{u'url': u'https://www.facebook.com/datomahsi...
[{u'contact_details': [], u'links': [], u'end_...
Mah Siew Keong
[]
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/api...
216
66
1949-03-01
[{u'type': u'phone number', u'id': u'557df20ec...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e45e35222837c2c05904e
...
NaN
[{u'url': u'https://www.facebook.com/DatukSeri...
[{u'contact_details': [], u'links': [], u'end_...
Palanivel A/L K. Govindasamy
[{u'name': u'Datuk G. Palanivel', u'id': u'558...
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/api...
217
57
1957-08-10
[{u'type': u'phone number', u'id': u'557c03bbb...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e48ea5222837c2c0597c7
...
NaN
[{u'url': u'https://twitter.com/mkula', u'note...
[{u'contact_details': [], u'links': [], u'end_...
M. Kulasegaran A/L V. Murugeson
[{u'name': u'M Kulasegaran', u'id': u'558f214b...
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/api...
218
61
1954-05-29
[{u'type': u'phone number', u'id': u'557df0bbc...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e47565222837c2c0593f2
...
NaN
[]
[{u'contact_details': [], u'links': [], u'end_...
Ong Ka Chuan
[{u'name': u'Dato' Seri Ong Ka Chuan', u'id': ...
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/api...
219
58
1957-01-27
[{u'type': u'phone number', u'id': u'5582a65e4...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e44855222837c2c058ce4
...
NaN
[{u'url': u'http://williamleongjeekeen.blogspo...
[{u'contact_details': [], u'links': [], u'end_...
William Leong Jee Keen
[]
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/api...
220
NaN
NaN
[{u'type': u'phone number', u'id': u'557df40fc...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e45635222837c2c058f0f
...
NaN
[]
[{u'contact_details': [], u'links': [], u'end_...
Abdul Rahman Bin Mohamad
[]
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/api...
221
NaN
NaN
[{u'type': u'phone number', u'id': u'5580ff404...
NaN
NaN
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/per...
545e46f25222837c2c0592f6
...
NaN
[{u'url': u'https://www.facebook.com/pages/Dat...
[{u'contact_details': [], u'links': [], u'end_...
Abdul Aziz Sheikh Fadzir
[{u'name': u'Datuk Abdul Aziz Sheikh Fadzir', ...
https://sinar-malaysia.popit.mysociety.org/ima...
NaN
NaN
NaN
https://sinar-malaysia.popit.mysociety.org/api...
222 rows × 23 columns
In [27]:
df = pandas.DataFrame(MPs)
print df['age'].median()
print df['age'].max()
print df['age'].min()
56.0
74.0
32.0
We could have dropped duplicates from bad data:
df.drop_duplicates('id')
Parse and set birth_date column as datetime to calculate age without parsing it manually:
df['birth_date']= pandas.to_datetime(df['birth_date'])
Best of all after cleaning up the data, we can easily export it to CSV format where it is more easily usable by normal users in spreadsheets or plotting charts.
In [28]:
MP_source = {'name':df['name'],'birth_date':df['birth_date'],'age':df['age']}
In [29]:
MP_Names = pandas.DataFrame(MP_source)
MP_Names.sort('age')
Out[29]:
age
birth_date
name
191
32
1982-10-17
Zairil Khir Johari
167
33
1982-05-13
Sim Chee Keong
65
34
1981-01-27
Teo Nie Ching
21
34
1980-11-19
Nurul Izzah Anwar
168
35
1979-08-09
Kasthuriraani Patto
171
37
1977-12-20
Ng Wei Aik
4
37
1977-09-14
Mohd Rafizi Bin Ramli
47
37
1977-11-27
Liew Chin Tong
43
37
1977-10-19
Chua Tee Yong
36
38
1976-11-13
Mas Ermieyati Samsudin
27
38
1977-04-28
Loke Siew Fook
178
39
1976-02-06
Sim Tze Tzin
30
39
1976-01-10
Khairy Jamaluddin Abu Bakar
0
39
1975-09-12
Ong Kian Ming
37
40
1974-12-29
Shamsul Iskandar Mohd Akin
53
42
1972-12-17
Er Teck Hwa
133
42
1972-12-28
Gooi Hsiao-Leung
6
42
1973-06-19
Gobind Singh Deo
205
42
1972-11-11
Nga Kor Ming
17
43
1972-07-25
Lim Lip Eng
76
43
1972-07-10
Rosnah Abdul Rashid Shirlin
8
43
1972-08-01
Tony Pua Kiam Wee
54
44
1971-07-24
Liang Teck Meng
98
44
1971-02-12
Chong Chieng Jen
42
44
1970-10-14
Razali Ibrahim
144
45
1970-03-22
Nik Mohamad Abduh Bin Nik Abdul Aziz
7
45
1970-06-18
Hee Loy Sian
82
45
1970-03-08
Raime Unggi
2
46
1968-12-18
Wong Chen
51
46
1968-10-20
Wee Ka Siong
...
...
...
...
156
NaN
NaN
Wan Abd Rahim Bin Wan Abdullah
160
NaN
NaN
Wan Hassan Mohd Ramli
161
NaN
NaN
Mohd Khairuddin Bin Aman Razali
164
NaN
NaN
Che Mohamad Zulkifly Bin Jusoh
165
NaN
NaN
Shabudin Yahaya
166
NaN
None
Ramkarpal Singh A/L Karpal Singh
170
NaN
NaN
Mansor Bin Othman
173
NaN
NaN
Jeff Ooi Chun Aun
174
NaN
NaN
Hilmi Bin Yahaya
176
NaN
NaN
Fuziah Binti Salleh
179
NaN
NaN
Mohd Ariff Sabri Bin Abdul Aziz
180
NaN
NaN
Ahmad Nazlan Bin Idris
181
NaN
NaN
Ismail Bin Abd. Muttalib
182
NaN
NaN
Shahril Azman Bin Abd Halim
184
NaN
NaN
Noh Bin Omar
187
NaN
NaN
Liow Tiong Lai
188
NaN
NaN
Irmohizam Bin Ibrahim
189
NaN
NaN
Nasrudin Bin Hassan
193
NaN
NaN
Su Keong Siong
194
NaN
NaN
Reezal Merican Bin Naina Merican
197
NaN
NaN
Hasbullah Bin Osman
198
NaN
NaN
N Surendran A/L K Nagarajan
201
NaN
None
Idris Ahmad
204
NaN
NaN
Sivakumar A/L Varatharaju Naidu
211
NaN
NaN
Madhi Bin Hasan
212
NaN
NaN
Mohd Zaim Bin Abu Hasan
214
NaN
NaN
Saravanan A/L Murugan
215
NaN
NaN
Mah Siew Keong
220
NaN
NaN
Abdul Rahman Bin Mohamad
221
NaN
NaN
Abdul Aziz Sheikh Fadzir
222 rows × 3 columns
In [30]:
%matplotlib inline
grouped = MP_Names.groupby('age')
grouped.age.count().plot(kind='bar',figsize=(15,15))
Out[30]:
<matplotlib.axes._subplots.AxesSubplot at 0x7f80c1142c90>
Previous example tried to massage python data structures into Pandas DataFrame which works, but isn't very pretty.
Ng Swee Meng sweester@sinarproject.org has contributed proper way of building up data structures for Pandas DataFrames in the following example:
In [32]:
import pandas
data = { "age": [], "birth_date": []}
data_index = { "age": [], "birth_date": []}
for entry in MPs:
if entry.has_key('age'):
data["age"].append(entry["age"])
data_index["age"].append(entry["name"])
data["birth_date"].append(entry["birth_date"])
data_index["birth_date"].append(entry["name"])
final_data = { "age": pandas.Series(data["age"], index=data_index["age"]),
"birth_date": pandas.Series(data["birth_date"], index=data_index["birth_date"])
}
mp_age_df = pandas.DataFrame(final_data)
mp_age_df.sort("age")
Out[32]:
age
birth_date
Zairil Khir Johari
32
1982-10-17
Sim Chee Keong
33
1982-05-13
Teo Nie Ching
34
1981-01-27
Nurul Izzah Anwar
34
1980-11-19
Kasthuriraani Patto
35
1979-08-09
Ng Wei Aik
37
1977-12-20
Mohd Rafizi Bin Ramli
37
1977-09-14
Liew Chin Tong
37
1977-11-27
Chua Tee Yong
37
1977-10-19
Mas Ermieyati Samsudin
38
1976-11-13
Loke Siew Fook
38
1977-04-28
Sim Tze Tzin
39
1976-02-06
Khairy Jamaluddin Abu Bakar
39
1976-01-10
Ong Kian Ming
39
1975-09-12
Shamsul Iskandar Mohd Akin
40
1974-12-29
Er Teck Hwa
42
1972-12-17
Gooi Hsiao-Leung
42
1972-12-28
Gobind Singh Deo
42
1973-06-19
Nga Kor Ming
42
1972-11-11
Lim Lip Eng
43
1972-07-25
Rosnah Abdul Rashid Shirlin
43
1972-07-10
Tony Pua Kiam Wee
43
1972-08-01
Liang Teck Meng
44
1971-07-24
Chong Chieng Jen
44
1971-02-12
Razali Ibrahim
44
1970-10-14
Nik Mohamad Abduh Bin Nik Abdul Aziz
45
1970-03-22
Hee Loy Sian
45
1970-06-18
Raime Unggi
45
1970-03-08
Wong Chen
46
1968-12-18
Wee Ka Siong
46
1968-10-20
...
...
...
Kamarudin Bin Jaffar
63
1951-11-03
Richard Riot Jaem
63
1951-12-01
Che Rosli Bin Che Mat
64
1951-07-16
Tengku Adnan Tengku Mansor
64
1950-12-20
Wahab Dolah
64
1950-11-15
Wong Sze Phin
65
1950-03-01
William Nyallau Badak
65
1950-07-22
Shahrir Abdul Samad
65
1949-11-22
Ahmad Lai Bujang
65
1949-11-26
Muhammad Leo Michael Toyad Abdullah
65
1950-04-11
Joseph Salang Gandum
65
1950-01-01
Palanivel A/L K. Govindasamy
66
1949-03-01
James Dawos Mamit
66
1948-11-04
Sim Tong Him
66
1948-09-08
Ahmad Hamzah
66
1948-09-27
Abdul Manan Ismail
67
1948-06-28
Tajuddin Bin Abdul Rahman
67
1948-01-17
Abdul Hadi Bin Awang
67
1947-10-20
Kamarul Baharin Abbas
68
1947-06-12
Abdul Khalid Ibrahim
68
1946-12-14
Muhyiddin Yassin
68
1947-05-15
Fauzi Bin Abdul Rahman
68
1946-08-27
Hasan Malek
69
1946-04-24
Wan Junaidi Tuanku Jaafar
69
1946-02-01
Henry Sum Agong
69
1946-02-18
Joseph Kurup
70
1944-12-15
Abdul Ghapur Salleh
72
1943-03-21
Tan Seng Giaw
73
1942-05-26
Joseph Pairin Kitingan
74
1940-08-17
Lim Kit Siang
74
1941-02-20
144 rows × 2 columns
In [34]:
mp_age_df["age"].plot(kind="hist",figsize=(15,15))
Out[34]:
<matplotlib.axes._subplots.AxesSubplot at 0x7f80c0a96710>
In [ ]:
Content source: lowks/sinar.myreps
Similar notebooks: